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ABSTRACT 

The Expert System Development Methodology (ESDM) provides an approach to developing 
expert system software. Because of the uncertainty associated with this process, an element of risk 
is involved. ESDM is designed to address the issue of risk and to acquire the information needed 
for this purpose in an evolutionary manner. ESDM presents a life cycle in which a prototype 
evolves through five stages of development. Each stage consists of five steps, leading to a 
prototype for that stage. Development may proceed to a conventional development methodology 
(CDM) at any time if enough has been learned about the problem to write requirements. ESDM 
produces requirements so that a product may be built with a CDM. ESDM is considered 
preliminary because it has not yet been applied to actual projects. It has been retrospectively 
evaluated by comparing the methods used in two ongoing expert system development projects that 
did not explicitly choose to use this methodology but which provided useful insights into actual 
expert system development practices and problems. This fiscal year, the methodology will be 
field-tested by applying it to two pilot projects. 

INTRODUCTION 

An expert system is a computer system that simulates, to the extent possible or practical, the 
way human experts solve cognitive problems, i.e., those for which no algorithm is known to solve 
the problem, but that are routinely solved by a domain expert using rules of thumb or heuristics. 

The task of developing an expert system is, therefore, to acquire information from the domain 
expert as to how the cognitive tasks are performed and then model the information in a form 
suitable for the computer. Because of the uncertainty associated with this process, an element of 
risk is associated with developing expert systems. ESDM has been designed to address the issue 
of risk and to acquire the information needed for this purpose in an evolutionary manner. 

Note that it is not possible to develop specifications for an expert system before the expert s 
reasoning processes have been analyzed. This lack of specifications presents the manager with 
special problems in controlling an expert system development project. These problems are 
addressed in ESDM. 

Analyses reveal differences between expert systems and conventional software development. 
Compared to conventional development, the expert system life cycle more nearly resembles rapid 
prototyping. The major differences between the expert system life cycle and the rapid prototyping 
of conventional development are 

• Greater uncertainty in feasibility and suitability of the problem to an expert system 
implementation 
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• Highly iterative nature of expert system development 

The major similarity is that the system produced is a prototype. More specifically, all 
systems produced by ESDM are considered prototypes. ESDM does not produce any of the 
standard products (e.g., feasibility study, requirements analysis) that are produced for a 
conventional software system. 

The methods used to implement expert systems differ from those used for conventional 
systems. However, once the uncertainty of the feasibility of an expert system is reduced to a low 
level (i.e., after several iterations of prototype development), it will be possible to undertake 
development of an expert system in a conventional development cycle and to expect it to present no 
more difficulty than any other system. ESDM precedes conventional development and terminates 
when requirements can be produced. 

ESDM is intended to be applied to the development of National Aeronautics and Space 
Administration/Goddard Space Flight Center (NASA/GSFC) expert systems. It is based on a 
survey of existing methodologies and an analysis of the expert system life cycle. Dr. Barry Boehm 
introduced a nsk-driven methodology for conventional system development in his spiral model for 
software development (1). ESDM, while independently generated, is also a risk-driven 
methodology that can be represented as a spiral model. ESDM focuses more on knowledge 
acquisition rather than on product development. Figure 1 illustrates the spiral nature of ESDM. 


PROTOTYPING STAGES 
(DIRECTION OF DECREASING OVERALL RISK) 



Figure 1. Spiral Model of ESDM 
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ESDM was developed to address large projects. Therefore, when applying ESDM to 
projects that are small, ESDM recommendations should be used as a guideline and adapted to suit 
the smaller-size project. 

The proposed methodology is considered preliminary because it is untested. It is a synthesis 
of methodologies that have been used in conventional and expert system software development. A 
standard (4), a user guide (6), a reference manual (3), and a set of training materials (5) have been 
developed to describe the methodology. The methodology has been retrospectively evaluated by 
comparing the methods used in two ongoing GSFC expert system development projects: the 
Ranging Equipment Diagnostic Expert System (REDEX) and the Backup Control Mode Analysis 
and Utility System (BCAUS) (2). These projects did not explicitly choose to use the methodology 
but provided useful insights into actual expert system development practices and problems. This 
fiscal year, the methodology is being field-tested by applying it to two other GSFC expert system 
development projects: the Generic Expert System and the Systems Test and Operations Language 
(STOL) Intelligent Tutoring System (ITS). Next fiscal year, the methodology will be revised 
based on the results obtained. 

DECISION POINTS OF ESDM 

ESDM consists of five stages, each accomplished in five steps, and is geared to reducing the 
uncertainty of feasibility and risk in development. The ESDM life cycle is illustrated in Figure 2, 
which shows five decision points that determine whether work must move to one or another of the 
five ESDM stages. In contrast with CDM, iteration of a stage or iteration of steps within a stage 
may also occur. (Iteration of steps is not shown explicitly in the figure.) Higher risk issues are 
addressed first, and lower risk issues are addressed with each subsequent stage. Earlier stages 
focus on knowledge acquisition, and later stages focus on performance issues. The stages and 
stage products described serve as a guide and should be adapted to suit the goals and objectives of 
each individual project. If an alternate area is of higher risk for the project, this area is prototyped 
first. 


In addition, there are three risk-based decisions, as follows: 

• Apply ESDM? Are expen system development techniques suitable for the 
problem? 

• Stop ESDM? Has enough been learned to decide whether to move to a CDM or 
to abort development? If the answer is no, then a stage is selected. A stage may be 
repeated or a new stage selected. If the answer is yes, the next decision is 
addressed. 

• Move to a CDM? Should development move to a CDM or be aboned? Are the 
requirements now known, or is the risk too high to continue? 

Development need not proceed through all five stages if these questions can be answered in 
an earlier stage. ESDM provides a metric— the Test for Application of Risk-Oriented Technology 
(TAROT)-to provide guidance to both managers and developers at each decision point. TAROT 
assists in evaluating a project for suitability to an expert system implementation and in estimating 
the risk involved. At the completion of the project using ESDM, a transfer to product cycle is 
undertaken and development continues using a CDM. 
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Figure 2. The ESDM Life Cycle 
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FIVE STAGES IN EXPERT SYSTEMS DEVELOPMENT 


The proposed ESDM consists of five stages of prototype development (8). Each stage 
consists of five steps (7). The stages are discussed in the following subsections; the steps for each 
stage are discussed in the next section. 

Stage 1: Feasibility 

Model one or more key cognitive functions performed by domain expert. 

A key cognitive function is one that is central to the overall problem the expert solves. 
Selecting a key function is important but not crucial because ESDM is highly iterative. 

Stage 2: Research (Extensibility) 

Model remaining functions. 

The goal of this stage is to determine the extent to which the remaining cognitive functions 
performed by the domain expert can be modeled. 

Stage 3: Field 

Model additional functions required in the field. 

This stage determines how to model the remaining cognitive functions performed by the 
expert. Specifically, this stage determines what combination of conventional and expert system 
techniques should be used to construct an automated replacement of the manual system. The goal 
of this stage is to design and construct a fieldable prototype that can be tested and used in a realistic 
setting. Performance issues have not been addressed yet. 

Stage 4: Production 

Address performance objectives. 

This stage determines if it is possible to refine the design of, recode, or transport the system 
to a new host to achieve the desired scope of expertise and performance objectives. The successful 
production prototype is able to solve nearly all required problems and is robust and is easy to use. 

Stage 5: Operational 

Analyze and evaluate risks and costs of deploying the production prototype. 

To reach this stage in ESDM, the development team has learned it is possible to build a 
prototype that simulates the functions performed by the domain expert A knowledge acquisition 
process is still involved in this stage, but the focus is now on the use of the product rather than on 
the structure of the product. 

FIVE STEPS IN EXPERT SYSTEMS DEVELOPMENT 

Within each stage of expert system development, five steps are required to produce the stage 
prototype. These steps are highly iterative and are not necessarily performed in a simple, 
sequential manner— developers often work on several steps concurrently. Only in a general sense 
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does the work progress from one step to the next. Typically, work can and does move from a later 
step to an earlier step as knowledge is acquired. 

Step 1: Identification 

Determine problem characteristics. 

In this step, the knowledge engineer identifies the problem the expert system will solve for 
the current stage. In the feasibility stage, the sources of expertise are identified. These sources 
may include one or more experts, reports, or manuals. Knowledge acquisition for the stage begins 
at this point 

Step 2: Conceptualization 

Find concepts to represent the knowledge. 

Knowledge acquisition continues in this step and includes identifying the concepts, objects 
the expert reasons about, the relationships between objects, control flow, constraints, and 
problem-solving strategies used by the expert. The knowledge is organized and modeled using a 
technique suitable to the problem (i.e., tables, informal rules, flow charts, hierarchies). 

Step 3: Formalization 

Map these concepts to formal representations based on the selected implementation tool or 
language. 

This step involves formalizing the concepts chosen in the conceptualization step, designing 
the system, and selecting the hardware and software tools for implementation. The suitability of 
the tool or language selected for representing the problem determines the difficulty of subsequent 
steps. 

Step 4: Implementation 

Formulate rules that embody knowledge. 

In the implementation step, a prototype is coded from the formal representations developed in 
the formalization step using the selected software tool or language. 

Step 5: Test 

Validate rules that embody knowledge and verify that the prototype implements the design. 

This step involves evaluating the prototype’s performance through testing. In the early 
stages of ESDM, the prototype’s performance is compared against that of the domain expert. In 
later stages, performance issues are evaluated (e.g., robustness, speed of execution, ease of use). 

TRANSFER TO PRODUCT CYCLE 

An ESDM project terminates and proceeds to CDM when the risk of feasibility and use are 
reduced to an acceptable level and when requirements can be sufficiently defined to continue 
development in a sequential manner. The findings of ESDM serve as the basis for CDM and are 
formalized and transferred in a formal transfer review meeting. Expert system project personnel 
should assist in, monitor progress of, or perform the conventional product development to handle 
problems or issues that might arise. 
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An ESDM project terminates and development is discontinued if the risks of either 
implementation or use are considered unacceptable. A final lessons learned review meeting is 
conducted to preserve knowledge acquired on the project and to explain this decision. 

THE ESDM PRODUCTS 

The products produced at the start of an ESDM project include the following: 

• Concept and Project Initiation Report — A high-level strategic plan for the 
project. It explains the results of initial risk and suitability analysis, describes the 
system to be automated, and justifies the project 

The products produced in ESDM in each stage of development include the following: 

• Stage Project Management Plan — A lower level, more detailed plan than the 
Concept and Project Initiation Report. It is produced at the start of each stage and 
describes work to be completed for that stage. 

• Knowledge Engineering Report — A summary of the knowledge acquired and 
the lessons learned during the stage, as well as recommendations for later stages of 
work. 

• Prototype Design Report — A detailed description of the rules or other 
knowledge structures used in the prototype. 

• Prototype Operations Guide — A description of prototype operation for 
subsequent testers. 

• Test and Evaluation Report — A description of the test results. Evaluates the 
prototype. 

• The prototype. 

The products produced at the end of a project include one of the following: 

• Technology Transfer Report — Produced following the decision to continue 
development with CDM. The report summarizes project findings and lessons 
learned, and presents recommendations for CDM development. 

• Project Termination Report — Produced following the decision to abort 
development The report justifies this decision, summarizes lessons learned, and 
documents results of knowledge acquisition. 
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